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^^^^^^^^^Accountability * Integrity * Reliability _ 

United States General Accounting Office 
Washington, DC 20548 


May 18, 2001 

The Honorable Thomas M. Davis 
Chairman, Subcommittee on Technology and 
Procurement Policy 
Committee on Government Reform 
House of Representatives 

The Honorable Vernon J. Ehlers 
Chairman, Subcommittee on Environment, 

Technology, and Standards 
Committee on Science 
House of Representatives 

The Honorable Constance A. Morelia 
House of Representatives 

This report conveys the results of our review of the Department of 
Commerce’s National Technical Information Service (NTIS). As you know, 
NTIS is a permanent repository and principal disseminator of scientific, 
technical, engineering, and business-related information. NTIS acquires 
such research reports primarily from federal agencies and their 
contractors and grantees as well as from international sources. 

As requested, our objectives were to determine (1) NTIS’ various 
functions; (2) the quantity, age, and demand trends of reports in NTIS’ 
repository; (3) the extent that reports in NTIS’ repository are readily 
available from other public sources; and (4) whether federal agencies are 
sending their reports to NTIS for sale to the public, as required by law, 
and, if not, the reasons for this lack of compliance. 

To determine NTIS’ functions and the quantity, age, and demand trends for 
reports in its repository, we interviewed agency officials and analyzed 
various reports. To determine the extent that research reports in NTIS’ 
repository were readily available from other public sources, we randomly 
selected 600 reports from the repository and searched for them on World 
Wide Web sites at the issuing organization and the Government Printing 
Office (GPO). 1 We also searched at FirstGov.gov and at a private search 


1 Includes GPO Access, the Federal Depository Library Program, and the GPO Bookstore. 
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Results in Brief 


engine called Google.com. We used our sample results to estimate what 
portion of the total population of reports in the repository was available at 
these Web sites. All estimates reported from the sample have sampling 
errors of plus or minus 10 percentage points or less. To determine whether 
agencies sent their reports to NTIS as the law requires, we decided that it 
was not practical for us to perform compliance audits at every major 
department and agency. Rather, we relied on a proxy for identifying 
agency noncompliance. We used reports that NTIS identified and acquired 
electronically from the issuing agencies’ Web sites during 2000 after the 
agencies did not provide NTIS with the reports in a timely manner. 2 
NTIS refers to this process as “harvesting” reports. In addition, we asked 
officials at the seven agencies with the most harvested reports why they 
had not sent their reports to NTIS. Appendix I contains a more detailed 
discussion of our scope and methodology. 

We conducted our review from September 2000 to April 2001, in 
accordance with generally accepted government auditing standards. 


In addition to its basic statutory clearinghouse repository function of 
collecting and disseminating full-text paper reports, NTIS provides various 
other fee-based services for agencies. These include brokerage services 
such as billing and collecting for other agencies that charge for products 
and services; distribution services such as disseminating another agency’s 
products or operating the Internal Revenue Service’s (IRS) Web site; and 
Web services such as FedWorld (an on-line information network). 

We estimate that NTIS has about 2.5 million reports 3 in its repository that 
are to be retained permanently. In recent years, NTIS’ repository has been 
growing at a slower pace. For example, the repository contains 56,000 
reports published in 1995 while only 10,000 reports published in 2000 had 
been added to the repository as of mid-November 2000. 

The newest reports in NTIS’ repository—those published in 2000— 
accounted for less than 1 percent of all research reports in its repository. 
We estimate that about 75 percent (1.8 million) are over 12 years old. Since 
1995, we estimate that NTIS has sold 1 or more copies of about 8 percent 


" We did not evaluate whether each report that NTIS harvested was required to be reported 
to NTIS by law. 

3 Includes reports that NTIS asks other agencies to provide directly to customers (i.e., 
source fills). 
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(187,000) of its 2.5 million reports. Of the 1.8 million reports over 12 years 
old (pre-1989), we estimate that only about 1 percent has sold since 1995. 

We estimate that about 19 percent (470,000) of NTIS’ reports were readily 
available from one or more of the four sources at the time we searched. Of 
these reports, we estimate about one-third could be obtained free from the 
Internet. The remaining 2 million reports (81 percent) were only readily 
available from NTIS. We estimate that reports published in 2000 were over 
five times more likely to be available at these other sources than those 
over 12 years old. 

Agencies often did not send their reports to NTIS as required by 15 U.S.C. 
3704b-2. In 2000, officials in NTIS’ Information Services Division harvested 
over 1,100 reports from agency Web sites that they believed should have 
been sent to NTIS but were not. These included reports from the 
Departments of Justice and Commerce, NTIS’ parent organization. The 
reasons agencies cited for not sending their reports included that they 
(1) were unaware of the law; (2) would incur additional costs and 
duplication of effort to format and transfer information to NTIS that is free 
on the agency’s Web site; or (3) did not believe their reports were covered 
because, in their view, their reports did not contain scientific, technical, or 
engineering information. 

The increasing availability of scientific, technical, and engineering 
information on the Web is raising fundamental policy questions about the 
future direction of NTIS. Therefore, we are presenting matters for 
consideration to the Congress on determining the best approach for 
defining, retaining, and disseminating government research in the Internet 
era. We are also making recommendations to the Secretary of Commerce 
to improve the extent to which agencies send their reports to NTIS, as 
required by law. 

In commenting on a draft of our report, the Secretary of Commerce agreed 
with our identification of the key policy issues and concurred with the 
thrust of our recommendations. The Secretary said he would determine 
how best to implement our recommendations and ask the Acting Under 
Secretary for Technology to determine if there are any practical ways, 
consistent with the statutory requirement that NTIS be self-sustaining, to 
make NTIS’ information collection more widely available to the general 
public. 
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Background 


Under statutory authority enacted in 1950, 4 NTIS is directed to collect 
scientific, technical, and engineering information from both domestic and 
foreign sources and to make the information available to (1) business and 
industry, (2) state and local governments, (3) other federal agencies, and 
(4) the general public to increase U.S. competitiveness in the global 
economy. The statute does not define scientific, technical, and engineering 
information. However, a 1954 Comptroller General Decision found the 
Secretary of Commerce’s position reasonable that the statute included “all 
types of information which have a more or less direct bearing on business 
and industry generally.” 5 The Secretary specifically described such 
information as including “economic information, market information, and 
related information so long as it is reasonably specific and bears some 
direct relationship to the organization and operation of industrial or 
business enterprise.” 

Agencies’ transfer of research information to NTIS is generally governed 
by 15 U.S.C. 3704b-2, which states that 

“The head of each Federal executive department or agency shall transfer in a timely 
manner to the National Technical Information Service unclassified scientific, technical, and 
engineering information which results from federally funded research and development 
activities for dissemination to the private sector, academia, State and local governments, 
and Federal agencies. Only information which would otherwise be available for public 
dissemination shall be transferred under this subsection. Such information shall include 
technical reports and information, computer software, application assessments generated 
pursuant to section 3710(c) of this title, and information regarding training technology and 
other federally owned or originated technologies.” 

This statute is implemented by NTIS regulation 15 C.F.R. Part 1180 (2000). 
The regulation, in general, provides for transfer to NTIS of (1) at least one 
copy of every final research product resulting from an agency’s federally 
funded research and development activities and (2) a summary of an 
agency’s new and ongoing research that is likely to result in a final 
product. 

NTIS’ basic statutory function is to collect research reports from agencies, 
maintain a bibliographic record and permanent repository of reports, and 
disseminate these reports to the public as required. NTIS charges user fees 


4 15 U.S.C. 1152. 

5 34 Comp. Gen. 58 (1954). 


Page 4 


GAO-01-490 NTIS Technical Reports 


for the sale of its products and services to the public. NTIS’ enabling 
legislation requires that, to the fullest extent feasible, all of the services 
and functions it provides be self-sustaining. 6 From fiscal years 1988 
through 1994, NTIS showed a net profit in all years except 1 and generated 
a cumulative net profit of $5.8 million. However, over the next 5-year 
period of fiscal years 1995 through 1999, NTIS had a cumulative net loss of 
about $4.2 million. 

In August 1999, the Secretary of Commerce proposed closing NTIS by 
September 30, 2000, because he believed that declining sales revenues 
soon would not be sufficient to recover all of NTIS’ operating costs. The 
Secretary partly attributed this decline to other agencies’ practice of 
making their research results available to the public for free through the 
Web. He also proposed transferring NTIS’ research report archives to the 
Library of Congress and requiring federal agencies to give the public free 
on-line access to new research reports. 

In June 2000, we issued a report noting that NTIS may be able to remain 
self-sustaining for a few years, but that its solvency beyond fiscal year 
2003 remained in doubt. 7 Since the issuance of our June report, however, 
NTIS’ Deputy Director said NTIS has significantly reduced costs and 
shown a profit of $2.2 million for fiscal year 2000, and the Deputy Director 
projects a profit of $1.5 million for fiscal year 2001. 


NTIS Provides a 
Variety of Services in 
Addition to Its 
Statutory Function of 
Collecting and 
Disseminating 
Research Reports 


In addition to NTIS’ basic statutory function of collecting and 
disseminating technical reports, NTIS also provides various other fee- 
based services for agencies. According to NTIS, these include brokerage 
services such as billing and collecting for other agencies that charge for 
products and services; distribution services such as disseminating other 
agencies’ products or operating the IRS’ Web site; and Web services such 
as FedWorld (an on-line information network). According to NTIS 
financial data, revenue from all functions totaled $35 million in fiscal year 
2000 . 

NTIS’ four major functions and the revenue they generated are described 
below with some examples: 


15 U.S.C. 1153. 

' Infor mation Policy: NTIS’ Financial Position Provides an Opportunity to Reassess Its 
Mission ICGAO/GGD-00-147J June 30, 2000). 
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Clearinghouse repository function. This is NTIS’ basic statutory function 
of collecting and disseminating technical reports, and the function 
includes seven product lines. According to NTIS, these product lines 
generated a total of almost $19 million (55 percent of NTIS’ total revenue) 
in fiscal year 2000: 

• Full-text products. This product line refers to NTIS’ basic document sales 
program. It includes reports that agencies and foreign sources provide 
either in hard copy or microfiche (retail); bulk sales to resellers 
(wholesale); and sales from special collections, such as military 
publications. It generated a reported revenue of about $6 million in fiscal 
year 2000. 

• Announcement products. This product line refers to information about 
new research reports available from NTIS. Included in this line are NTIS’ 
bibliographic database, which is disseminated to the public for a fee by 
vendors who share the resulting revenue with NTIS, and “NTIS Alerts,” a 
biweekly product announcing the availability of new products in specific 
subject categories. It generated a reported revenue of about $1.2 million in 
fiscal year 2000. 

• Computer products. This product line includes all sales of software, CD- 
ROMs, and diskettes—containing such items as export regulations, IRS 
forms, and IRS’ Small Business Resource Guides. It generated a reported 
$5 million in fiscal year 2000. 

• Paper subscription products. These are requests by users for a specific 
information product that is issued or updated on a regular, predictable 
basis and in hard copy. Examples of these products include Foreign 
Agricultural Service publications and EPA Quarterly Abstracts. The 
product line generated a reported $890,000 in fiscal year 2000. 

• Electronic subscription products. These are products that are 
disseminated electronically on a predictable basis, such as World News 
Connection (an on-line foreign news service), Federal Research in 
Progress, and Davis-Bacon Act wage determinations. The product line 
generated a reported $3 million in fiscal year 2000. 

• Standing order products. These are similar to subscriptions, except that 
the product is issued or updated on an irregular basis, such as National 
Transportation Safety Board Accident Reports. The product line generated 
a reported revenue of almost $2 million in fiscal year 2000. 
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• Audiovisual products. In addition to paper and electronic formats, NTIS 
sells a number of audiovisual products, such as videocassettes, that 
contain training, health and safety, environmental, and other forms of 
information. These products are collected by NTIS’ audiovisual center, 
which was transferred to NTIS from the National Archives and Records 
Administration. The product line generated a reported $590,000 in fiscal 
year 2000. 

Brokerage services. NTIS provides billing and collection services on a 
reimbursable basis to other agencies that, like NTIS, charge for products 
and services, but lack the necessary financial infrastructure to do their 
own billing and collecting. The product line generated about $340,000 in 
fiscal year 2000, according to NTIS officials. 

Distribution services. As an outgrowth of its own activities to disseminate 
research reports, NTIS will also produce and/or disseminate specific 
information products to an agency’s constituents under a reimbursable 
agreement with the agency, such as operation of the IRS’ Web site. These 
products may be distributed in hard copy, on videocassette, on CD-ROM, 
or via facsimile or the Web. The agency may supply a mailing list or direct 
its constituents to call NTIS. NTIS may provide related “help desk” 
services to assist those constituents in using the material. At times, NTIS 
will develop multiformat packages for an agency, such as videocassettes 
and printed instructional material for use in training courses. In addition, 
NTIS offers agencies comprehensive services to integrate hardware and 
software in the design of systems that allow agencies to conduct 
transactions over the Internet. According to NTIS, distribution services 
generated about $15 million in fiscal year 2000. 

Web services. NTIS’ statutory charter calls for it to develop new methods 
or media for disseminating information; in 1992, the statute was amended 
specifically to include electronic media. In response, according to NTIS 
officials, NTIS developed the FedWorld platform as a means of handling 
high-volume traffic over the Internet in a secure environment—with 
sufficient capacity to allow NTIS to offer it for a fee to other agencies 
having a similar need. Web services generated a reported revenue of about 
$660,000 in fiscal year 2000. 

According to NTIS, in addition to the four major functions, it provides 
other miscellaneous services, such as experimenting with new ways to 
help agencies reach their constituents. For several years, NTIS has offered 
CD-ROM mastering and premastering services. Recently, NTIS and the 
Department of the Navy developed a program that provides sophisticated 
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satellite broadcast services to agencies that need to reach employees or 
constituents around the country or the globe. 


Quantity, Age, and 
Demand Trends 
of Reports in NTIS’ 
Repository 


We estimate that NTIS’ repository has about 2.46 million reports— 

2.04 million recorded in a database 8 and 420,000 older reports that were 
manually recorded on index cards. Of these estimated 420,000 older 
reports, we estimate that 200,000 were maintained and disseminated by 
other agencies on behalf of NTIS. 

Our estimate of 2.46 million reports is somewhat lower than the nearly 
3 million reports NTIS stated in its Web site and financial audit report for 
fiscal year 2000. NTIS overstated its holdings because it (1) double 
counted reports that were in both its database and on index cards and 
(2) overstated the number of reports maintained and disseminated by 
other agencies on behalf of NTIS (i.e., “source fill” reports). 

As shown in figure 1, we estimate that about 75 percent of the reports in 
NTIS’ repository were over 12 years old (i.e., published in 1988 or earlier) 
The most recent reports—those published in 2000—accounted for less 
than 1 percent of all reports in NTIS’ repository. Specifically, there were 

10,000 reports published in 2000, 

97,000 reports published from 1997 to 1999, 

516,000 reports published from 1989 to 1996, and 
an estimated 1,835,000 9 reports published in 1988 or earlier. 


8 The automated file we analyzed was as of mid-November 2000. 

9 This figure is an estimate because it includes our estimate of the number of reports 
recorded on index cards plus the number recorded in the database over 12 years old. 
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Figure 1: Percentage of Reports, by Year of Publication 


4% 

2000 



0.4% 

1988 or earlier 


1989-96 


1997-99 


Source: NTIS. 

As shown in figure 2, in recent years, NTIS’ repository has been growing at 
a slower pace—from about 56,000 reports published in 1995 to just over 
10,000 published in 2000 (as of mid-November 2000). 10 However, the 
Assistant to the Director of NTIS said that since about 75 to 80 percent of 
the reports in NTIS’ collection for a given year are added in the years 
following the year of publication, the final number of reports for 2000 
might increase significantly. Nevertheless, the Assistant to the Director of 
NTIS acknowledges that the total number of reports added to the 
repository in recent years has been declining. 


10 These data do not include reports in which NTIS (1) has only the bibliographic record 
(not the report) and (2) refers customers to the source agency for copies. For example, 
NTIS began making available military specifications and standards in 1997 and 1998 as part 
of a joint venture with the private sector, but NTIS is no longer providing these data to 
customers. 
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Figure 2: Year of Publication for Full-Text Reports in NTIS Repository 

Number of reports 
80000 



Calendar year 

Source: NTIS. 

Although total demand for NTIS reports has been increasing since fiscal 
year 1996, the public’s demand for NTIS paper reports has declined, while 
demand for electronic products increased, as shown in figure 3. 
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Figure 3: Products Distributed 
Numbers in millions 



Fiscal year 


I I Paper 

I I Electronic 

Source: NTIS. 


Demand for products was primarily from business and industry sources. 
As shown in table 1, nearly three-fourths of NTIS’ sales revenues during 
the last 3 fiscal years from the clearinghouse (except subscription sales) 
came from business and industry. The next largest customer segments 
were schools/universities, the federal government, and state and local or 
foreign governments. 
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Table 1: NTIS Sales, by Customer Segment 

Dollars in thousands 

Customer segment 

FY 1998 

FY 1999 

FY 2000 

3-year average 
as percentage 
of total revenue 

Business/Industry 

$11,045 

$10,813 

$10,257 

74% 

Schools/Universities 

969 

864 

857 

6 

Federal government 

755 

1065 

538 

5 

State, local, and 

883 

785 

679 

5 

foreign governments 





Individuals 

671 

710 

912 

5 

Foreign dealers 

293 

274 

279 

2 

Nonprofit 

242 

254 

248 

2 

Total 

$14,858 

$14,765 

$13,770 

99% a 


“Does not add to 100 percent due to rounding. 
Source: NTIS. 


Fewer Than One in Ten Since 1995, we estimate that NTIS has sold one or more copies of about 

Reports in Repository Has 187,000 (8 percent) of its 2.46 million reports. 11 As shown in figure 4, a 

Sold Since 1995 higher percentage of reports published from 1989 to 1999 have sold than 

the newest reports or the oldest reports. Of the 1.8 million reports over 
12 years old (pre-1989), we estimate that only about 1 percent has sold 
since 1995. 


11 This is the number of reports (i.e., titles) that sold one or more copies, not the total 
number of copies sold, which would be greater. We used 1995 because that is the year NTIS 
began recording sales data in its database. A report that was published in 1995 has had 6 
years to sell while a report published in 2000 has had 1 year to sell. 
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Figure 4: Estimated Percentage of Reports Sold in Each Stratum 
Percentage 



Year of publication 

I I Not sold 

I I Sold 

Source: GAO estimates from NTIS data. 


NTIS’ Best Sellers 
Comprise a Variety 
of Reports on Various 
Topics 


The Assistant to the Director of NTIS described “best sellers” as products 
that have a broad audience appeal or that sometimes are targeted for a 
particular audience. These are products for which NTIS does special 
product development, marketing, and promotion activities to enhance 
distribution. NTIS officials sometimes work with agencies and its 
contractors to preserve or resurrect a popular publication that the agency 
can no longer afford to produce on its own. NTIS also helps an agency 
customize its product for particular audiences, such as the medical 
community. 
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According to NTIS data, revenue from the sale of only the best sellers in 
fiscal year 2000 totaled over $1 million 12 —about 6 percent of the total 
$19 million in revenue from the sale of all paper and electronic products in 
the repository that year. 

NTIS’ best-selling products comprise a variety of reports on various topics. 
The following report titles are examples of those that appeared as best 
sellers on NTIS’ Web site in January 2001: 

“Any PLA (Primary Insurance Amount) - A Program to Estimate Social Security Monthly 
Benefit Payments”—Explains how much Social Security benefits will be at retirement. 

“Federal Tax Products from the IRS”—Includes forms, publications, and bulletins. 

“Statistical Abstract of the United States”—Reference book containing vital statistics on 
the United States. 

“US Industry and Trade Outlook”—Business forecasts for over 200 industries. 

“Armed Forces Recipe Cards”—1,300 recipes for groups of 100 that can be adjusted up or 
down. 

According to the Assistant to the Director of NTIS, these best sellers are 
usually sold in one or more of the following formats: 

Published books. These books are either printed at GPO (although not 
necessarily distributed by GPO) and purchased by NTIS or developed by 
NTIS and a joint venture partner. According to NTIS records, these books 
generated almost $700,000 in sales in fiscal year 2000. 

CD-ROM. In most cases, these NTIS products are developed from 
published books as an alternative format and are available only from NTIS. 
The one major exception is the IRS CD-ROM. According to NTIS records, 
these products generated over $200,000 in sales in fiscal year 2000. 

Audiovisual. These products cover a variety of topics. For example, NTIS 
sells a 21-minute tape called “Ergonomic Programs That Work” for $55.00 


12 In addition, sales of IRS CD-ROM tax products (forms, publications, and bulletins) were 
$1.3 million. We did not include these sales in the best-seller totals because NTIS officials 
said this is a unique product that NTIS bids for and may not always be able to offer. 
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from the Occupational Safety and Health Administration. According to 
NTIS records, these generated about $136,000 in sales in fiscal year 2000. 


Nineteen Percent of 
the Reports in NTIS’ 
Repository Were 
Available From Other 
Public Sources 


We estimate that about 470,000 of NTIS’ 2.46 million reports (19 percent) 
could be readily obtained through one or more of the four sources at the 
time that we searched. 13 The remaining 2 million reports (81 percent) 
could be readily obtained only from NTIS. 

As shown in figure 5, the same NTIS reports often were available at more 
than one of the four Web sites. For example, of the 10,000 reports 
published in 2000, we estimate that 56 percent were available from the 
issuing organization, 40 percent from sources identified through 
Google.com, 17 percent from sources identified through FirstGov.gov, and 
6 percent from GPO. For reports in all age categories, the source with the 
greatest probability of having the NTIS report was the issuing 
organization, and the source with the least probability was GPO. For 
reports published from 1989 to 2000, Google.com was able to locate more 
than twice the percentage of reports than either FirstGov.gov or GPO’s 
Web site search engines could locate. 


13 For this estimate, we counted reports available at more than one Web site only once. Had 
we not known the complete title of the report and its date of publication as shown in NTIS’ 
database, we might not have found as many reports. 
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Figure 5: Estimated Report Availability at Different Sources 

Percentage available 
60 



Year of publication 


I I GPO 

I | FirstGov.gov 

H Google.com 
I Agency Web site 

Source: GAO estimates from Web site data. 


As shown in figure 6, reports published in 2000 were over five times more 
likely to be available elsewhere than reports published in 1988 or earlier, 
and the percentage not readily available from these sources increased with 
the age of the report. For example, we estimate that only 12 percent of the 
1.8 million reports published in 1988 or earlier were readily available from 
the other sources. 
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Figure 6: Estimated Reports Available, by Year of Publication 

Percentage 

100 

90 



Year of publication 

I I Available 

I I Not available 

Source: GAO estimates from Web site data. 


Thirty-Seven Percent of 
NTIS’ Reports 
Available Elsewhere Were 
Free From the Internet 


We estimate that about 173,000 (37 percent) of the nearly 470,000 NTIS 
reports available elsewhere could be obtained for free from one or more of 
the four Internet sources we searched. For example, the following reports 
available for a charge from NTIS, were free from the issuing organization’s 
Web site: 


“Arizona - 1997 Census of Agriculture State and County Data. Volume 1. Geographic Area 
Series Part 3,” March 1999, National Agricultural Statistics Service, 328 pages. (At NTIS, a 
paper copy costs $25, microfiche $23.) 

“Packaging - and Transportation-Related Occurrence Reports - 1993 Annual Report,” June 
1994, Oak Ridge National Laboratory, Department of Energy, 116 pages. (At NTIS, a paper 
copy costs $36.50, microfiche $21.) 

“Consumer Use of Information - Implications for Food Policy,” June 1999, Economic 
Research Service, 32 pages. (At NTIS, a paper copy costs $14, microfiche $12.) 
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“Summary Travel Characteristics, District of Columbia. 1995 American Travel Survey,” 
October 1997, Bureau of Transportation Statistics, 26 pages. (At NTIS, a paper copy costs 
$23, microfiche $12.) 

As shown in figure 7, reports published from 1997 to 2000 were 
considerably more likely to be available free on the Internet than were 
reports published in other years. 


Figure 7: Estimated Percentage of Reports Available Free From Other Sources 

Percentage 

50 



Year of publication 

Source: GAO estimates that were based on Web site data. 


Reports That NTIS Had 
Sold Since 1995 Were More 
Readily Available 
Elsewhere Than Those Not 
Sold 


As shown in figure 8, we found reports that NTIS had sold since 1995 were 
more likely to be available elsewhere than reports that had not sold. 
Specifically, we estimate that 44 percent of the reports that sold in all age 
strata since 1995 were readily available elsewhere, compared with only 
17 percent of the reports that did not sell since 1995. 
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Figure 8: Estimated Availability of Reports Sold Since 1995 
Percentage 
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Source: GAO estimates that were based on Web site data. 



Agencies Did Not 
Always Send Their 
Reports to NTIS, as 
Required by Law 


Executive branch agencies are required by 15 U.S.C. 3704b-2 to transfer to 
NTIS unclassified scientific, technical, and engineering information 
resulting from federally funded research and development activities. NTIS’ 
implementing regulation 15 C.F.R. Part 1180 (2000) 14 defines scientific, 
technical, and engineering information as (1) basic and applied research 
that results from the efforts of scientists and engineers in any medium 
(including new theory and information obtained from experimentation, 
observation, instrumentation, or computation in the form of text, numeric 
data, or images) and (2) information that bears on business and industry 
generally, such as economic information and market and related 
information, if the agency determines such information would be of value 


14 NTIS’ regulation defines the term “agency” as used in the Stevenson-Wydler Act, which 
specifically includes legislative branch agencies. Section 3704b-2, however, is applicable 
only to “each federal executive department or agency,” excluding legislative branch 
agencies. Accordingly, we brought this issue to the attention of NTIS officials, and they told 
us they would consider an appropriate amendment to the regulation. 
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to consumers of the information described in the first part of this 
definition. The regulation requires agencies to send NTIS at least one copy 
of every final research product and further provides that agencies can 
enter into agreements with NTIS to transfer reports using alternative 
means. Such alternative means include providing the location for reports 
on the agency’s Web site so that NTIS can review the reports and select 
ones it believes meet the regulation’s requirements. 

However, according to NTIS, agencies are not providing all of the required 
reports. To obtain more of those reports, in 2000, NTIS began harvesting 15 
electronic versions of reports from agency Web sites. Although it would be 
difficult to identify the universe of federal reports that agencies should 
have provided to NTIS, harvesting provides some perspective on the 
extent to which agencies are complying with the regulation’s intent. For 
example, in 2000, an employee in NTIS’ Information Services Division 
harvested 1,127 reports 16 from agency Web sites that NTIS believes fit the 
definition in their regulation and should have been provided. NTIS 
harvested 10 or more of the 1,127 reports from each of the following 
agencies’ Web sites: 

• Federal Reserve System (169 reports) 

• International Trade Commission (151) 

• Office of National Drug Control Policy 17 (108) 

• Commerce’s International Trade Administration (95) 

• Department of Transportation’s Federal Highway Administration (61) 

• Commerce’s Bureau of the Census (52) 

• Department of Justice (41) 

• Federal Emergency Management Agency (24) 

• Federal Communications Commission (20) 

• Department of Energy’s Energy Information Administration (18) 

• Agency for International Development (18) 

• The White House (13) 

• Department of the Interior’s Fish and Wild lif e Service (11) 

• Department of Agriculture’s Forest Service (10) 


15 Harvesting is NTIS’ process of searching agency Web sites and downloading reports that 
it believes should have been submitted by the agencies for inclusion in NTIS’ repository. 

16 These reports included 19 GAO reports. GAO, as a legislative branch agency, is not 
required to submit reports to NTIS. 

17 Office of National Drug Control Policy (ONDCP) officials questioned whether ONDCP— 
as an agency of the Executive Office of the President—is covered by the law. 
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Commerce’s National Institute of Standards and Technology (10) 
Federal Trade Commission (10) 


The seven agencies with the most harvested reports had one or more of 
three general reasons for not sending their reports to NTIS: 

• they were not aware of the law (2 agencies); 

• they would incur additional costs and duplication of effort to format and 
transfer the information to NTIS that is free on the agency’s Web site 

(4 agencies); and/or 

• they thought that a few of the reports harvested by NTIS were not 
scientific, technical, or engineering reports and, thus, did not fall within 
the law’s scope (3 agencies). 

For example, Justice officials did not think that their harvested reports 
School Shooter: A Threat Assessment Perspective and Reducing Illegal 
Firearms Trafficking: Promising Practices and Lessons Learned were 
covered because they were not scientific, technical, or engineering 
reports. These officials said that additional guidance was needed on the 
type of reports covered under the law. Similarly, Transportation did not 
think that its Inspector General’s strategic plan was covered under the law. 
It is unclear whether reports like these fall under the definition and should 
be submitted to NTIS. However, our cursory review of the titles of the 
1,127 harvested reports found that most, although not all, appeared to 
contain scientific, technical, engineering, or business-related information. 

NTIS has taken steps to improve compliance and thereby reduce the need 
for harvesting. The Assistant to the Director of NTIS said that its Office of 
Agency Liaison (1) helps agencies identify the kind of material they should 
be sending and (2) reminds agencies of their legal requirements, although 
NTIS’ efforts are limited by available resources. In addition, at one time, 
NTIS and Energy had an agreement—now expired—that NTIS would 
harvest those reports it required directly from Energy’s Web site, rather 
than Energy’s sending NTIS the reports. The Assistant to the Director of 
NTIS said that this arrangement saved NTIS the costs of scanning paper 
reports and simplified the agency’s responsibilities under the regulation. 

He also noted that Energy made harvesting easy by systematically 
collecting reports from the various Energy offices and components at one 
location on its Web site. This agreement was not renewed, however, when 
NTIS and Energy could not agree on the specific terms under which they 
would continue this arrangement. 
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NTIS, however, has not pursued other ways of improving compliance. 
Although NTIS found the agreement with Energy to be beneficial, NTIS 
has no additional agreements in place. NTIS does provide instructions to 
agencies on alternative means for transferring reports—such as identifying 
the location of reports on agency Web sites or submitting by other 
electronic formats—but only when requested. Further, the Assistant to the 
Director of NTIS stated that NTIS does not routinely notify agencies that it 
harvested their reports and/or discuss with them why these reports were 
harvested. 

As a way of keeping the Congress informed on whether agencies are 
sending their technical reports to NTIS, the Secretary of Commerce is 
required, under subsection (b) of 15 U.S.C. 3704b-2 to report annually to 
the Congress “on the status of efforts under this section to ensure access 
to federal scientific and technical information by the public.” The Assistant 
to the Director of NTIS said that (1) they have not submitted any of these 
required reports because they understood the Congress and the executive 
branch were trying to eliminate surplus reporting requirements and (2) 
NTIS had erroneously assumed that their annual financial statement audit 
reports would include the necessary information to meet this requirement. 


Improving 
Compliance Would 
Not Diminish the 
Need to Address 
Fundamental Policy 
Issues 


Attempting to achieve greater compliance with the requirement for 
agencies to send research reports to NTIS is appropriate under the 
government’s current strategy for disseminating scientific, technical, and 
engineering information. However, the rising demand for electronic 
products, coupled with increasing availability of this information on the 
Web, raises more fundamental issues about how the information should be 
collected, stored, and disseminated—and, specifically, about the future of 
NTIS itself. These issues include whether the government’s current 
approach—a central repository to provide permanent public access on a 
self-sustaining basis—is still appropriate in the current environment. 


In recognition of this and in response to a request from several Members 
of Congress, the National Commission on Libraries and Information 
Science (NCLIS)—an independent agency that provides advice to the 
President and the Congress on information issues—conducted a study of 
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public information dissemination, which included an assessment of NTIS. 
NCLIS’ final report was issued in January 2001. 18 

Specifically, the policy issues raised by NCLIS and our studies include the 
following: 

How should “scientific, technical, and engineering information” be defined 
in the current environment? NTIS has defined scientific, technical, and 
engineering information broadly to include information directly related to 
business and industry that results from the efforts of scientists and 
engineers. NTIS, as the agency responsible for administering the law, is 
given discretion to define such terms. However, this broad definition has 
led to the agency’s acquiring and disseminating publications that appear to 
bear little relationship to business and industry, such as some of the “best 
sellers.” This raises the question of whether policymakers believe that the 
mission of NTIS should be to disseminate this variety of documents or 
whether a narrower interpretation of scientific, technical, and engineering 
information would be more appropriate in the current environment. 

NCLIS also addressed this issue. According to NCLIS, NTIS—in an effort 
to find revenue to support its operations—has expanded the scope of its 
collection well beyond its primary mission of collecting, organizing, and 
disseminating scientific and technical information. NCLIS concluded that 
NTIS’ scope should not be restricted to science and technology narrowly 
defined. However, they added that the scope should not include general 
public information that does not have a strong and direct relationship with 
business, industry, or technology. 

Is there a need for a central repository to disseminate scientific, technical, 
and engineering information? The increasing availability of newer 
publications from sources other than NTIS, along with the relatively low 
demand for older reports, may raise the question of whether a central 
repository for scientific, technical, and engineering information is still 


18 A Comprehensive Assessment of Public Information Dissemination, National 
Commission on Libraries and Information Science, January 26, 2001. 
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needed. 19 Answering this question requires knowing (1) the extent to 
which agencies could be relied upon to provide permanent public access 
to publications and (2) in the absence of a central repository, what type of 
bibliographic information and control over government publications are 
needed to ensure that reports can be identified and located indefinitely. 

In its study, NCLIS stated that the notion of a central source for 
government technical information to the public should not be abandoned. 
NCLIS, however, recommended consolidating the missions, functions, and 
programs of NTIS—along with other government entities responsible for 
disseminating government information to the public—into new 
independent agencies in each branch of the government. 

What obligations should a central repository have to retain all reports 
indefinitely? We estimate that only 1 percent of the NTIS reports over 
12 years old has sold since 1995. This information suggests that it may be 
useful to consider whether all scientific, technical, and engineering 
information should be retained permanently. How long to retain such 
information, however, would have to be based on demand patterns for 
older documents and on the costs and benefits of retaining documents of 
various age strata. This decision should also be based on a policy 
determination about permanent access and the need to maintain a central 
repository, as opposed to reliance on other repositories such as libraries. 

The NCLIS study also addressed the issue of permanent access to federal 
information. The study concluded that permanent public availability of 
and access to government information are critical parts of the overall 
strategy to meet the public information needs of the public and need to be 
strengthened. Further, the study concluded that NTIS is the only fail-safe 
source to make research results available when they are no longer 
available from the originating agency. 


19 In our report Major Management Challenges and Program Risks: A Governmentwide 
Perspective (GAO-Ol-241, January 2001), we suggested that the broad-based and rapid 
changes facing government today call for a fundamental review of existing programs and 
operations that address the following questions: (1) Do the conditions that caused the 
program to be created still exist? (2) Would we design the same program in the same way 
if we were designing it today? (3) Are there other government programs or policies that 
are not coordinated with or are working at cross-purposes to this program? (4) Is the 
program operating in the most economical, efficient, and effective manner? (5) Is the 
program affordable and sustainable both now and over the longer term? and (6) Is the 
program targeted appropriately to the beneficiaries most in need of federal assistance? 
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Conclusions 


If NTIS continues to exist as a central repository, what should be its 
business model? Issues concerning the financial viability of NTIS and the 
availability of research reports on agency Web sites raise questions about 
the appropriateness of NTIS’ business model. NTIS currently is required to 
be self-sustaining while maintaining permanent public access to scientific, 
technical, and engineering information in their repository. 

In its report, NCLIS recommended one such alternative business model. 
Specifically, NCLIS identified a number of NTIS functions that they believe 
generally benefit the public and recommended that these functions be 
funded with appropriations. These so-called “public good” functions 
included (1) collection or acquisition of reports, (2) cataloging and 
indexing, and (3) offering the NTIS database and full text of reports—to 
the extent that they are not available on agency Web sites—on NTIS 
servers for free public access. 


In addition to its statutory function of being a permanent repository and 
disseminator of scientific, technical, and engineering information, NTIS 
provides a variety of other fee-based services for agencies. In NTIS’ 
permanent repository of about 2.5 million reports, the number being added 
has been declining in recent years, and about 75 percent of the reports are 
now over 12 years old. While demand for electronic NTIS products is 
rising, research reports and other scientific, technical, and engineering 
information maintained by NTIS are also increasingly available on agency 
Web sites and through other public sources—often at no cost. 

Further, although NTIS has taken action to encourage agency compliance 
with the requirement to send it their research reports, agencies are often 
unaware of the law or confused about its application. NTIS has not been 
successful in pursuing agreements with agencies to facilitate the transfer 
of required documents. Finally, NTIS has not submitted its required annual 
report on the status of its efforts to ensure public access to scientific and 
technical information. As a result, NTIS has missed an opportunity to 
inform the Congress of the challenges that it faces. 

Broader policy questions remain to be answered about how government 
information should be disseminated in this Internet era and about NTIS’ 
role, if any, in that effort. 
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Matters for 

Congressional 

Consideration 


In light of developments in report dissemination through the Internet, the 
Congress may wish to reexamine how scientific, technical, and 
engineering information is disseminated to the public. In this 
reexamination, the Congress could specifically look at how this 
information is defined; whether there is a need for a central repository of 
this information, such as at NTIS; and, if a central repository is maintained, 
whether ah information should be retained permanently, and what 
business model should be used to manage it. 


Recommendations for 
Executive Action 


In the interim, while the Congress is reexamining the broader policy issues 
of the dissemination of government research information, we recommend 
that the Secretary of Commerce instruct the Director, NTIS, to 


• contact the major departments and agencies to outline the law’s 
requirements for sending reports to NTIS and offer assistance in helping 
these organizations comply; 

• pursue agreements with agencies to facilitate harvesting reports from 
agency Web sites as an alternative method of achieving compliance; 

• establish, in consultation with the President’s Council on Integrity and 
Efficiency, a method for monitoring future compliance, which could 
involve the participation of agency inspectors general; and 

• submit annual reports to the Congress, as required, on the status of actions 
taken to ensure access by the public to federal scientific, technical, and 
engineering information, including any action taken to implement these 
recommendations. 


Agency Comments 


We provided a draft of this report to the Secretary of Commerce for review 
and comment. The Secretary’s comments are reprinted as appendix II. 

In a letter dated April 26, 2001, the Secretary of Commerce agreed with our 
identification of the key policy issues—how to define scientific and 
technical information, whether there is a need for a central repository, 
how long reports should be maintained, and what is the proper business 
model for NTIS. The Secretary stated that, of these issues, the key 
question is whether there is a need for a central repository in view of the 
increasing availability of newer publications from sources other than 
NTIS. The Secretary also noted the need for a central repository depends 
on whether the information will be permanently maintained by agencies 
and whether it will be easy to locate absent the kind of bibliographic 
control NTIS provides. 

The Secretary concurred with the thrust of our recommendations and said 
he would determine how best to implement them. He said he will ask the 
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Acting Under Secretary for Technology to determine if there are any 
practical ways, consistent with the statutory requirement that NTIS be 
self-sustaining, to make NTIS’ information collection more widely 
available to the general public. 


As agreed with your offices, unless you publicly announce the contents of 
this report earlier, we will not distribute it until 30 days from the date of 
this report. At that time, we will send copies to the Honorable Donald L. 
Evans, the Secretary of Commerce; the Honorable Mitchell E. Daniels, Jr., 
Director, OMB; the Honorable Michael F. DiMario, the Public Printer; Mr. 
Ronald Lawson, Director, NTIS; and other interested congr essional 


committees. Copies will also be available at our Web site at lwww.gao.gov. 


Should you have any questions on matters contained in this report, please 
contact me at (202) 512-6240 or Alan Stapleto n, Assistant Direct or, at (202) 


512-3418. We can als o be reached by e-mail at |koontzl@gao.gov 


or 


ptapletona@gao.govJ respectively. Other key contributors to this report 
included Wendy Ahmed, William Isrin, Michael Jarvis, Frank Maguire, 
Warren Smith, and Ray Wessmiller. 


voC<K, 

Linda D. Koontz 

Director, Information Management Issues 
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Appendix I: Scope and Methodology 


To identify the functions that NTIS performs, we interviewed National 
Technical Information Service (NTIS) officials and reviewed various 
reports and studies, including NTIS’ handbook NTIS Services for Federal 
Agencies. 

To determine the quantity of NTIS’ research holdings, we added the 
number of newer reports recorded in NTIS’ “STAR” database to the 
estimated number of older reports recorded on index cards in NTIS’ file 
drawers. To estimate the number of reports recorded on index cards, we 
multiplied the number of drawers times the average number of cards in 
selected drawers to arrive at our initial estimate of 1.5 million reports. 

To determine if the data in STAR were valid and reliable, we 

• randomly selected 59 accession numbers from the total recorded in STAR 
and attempted to locate these reports in NTIS to verify that the title and 
date of issuance were accurate and 

• randomly selected 59 reports that were located near (above, below, to the 
right, or to the left) of the 59 reports to verify that they were recorded in 
STAR or in the index card drawers, depending on when the reports were 
issued. 

NTIS officials said that old reports were recorded on index cards located 
in hundreds of drawers. To verify that NTIS had the reports recorded on 
these cards and that the titles and dates were accurate, we randomly 
selected 75 cards and asked NTIS to attempt to locate these reports. These 
same 75 cards were part of our random sample of 150 reports 12 years or 
older. 

We found that NTIS’ computer generated data were sufficiently valid, 
complete, and reliable for purposes of our review, but the data in the index 
card drawers were not reliable. In particular, we found that 

• the reports were available and the titles were accurately recorded in STAR 
for all 59 randomly selected accession numbers; the year of the report’s 
issuance was accurately recorded for 54 of the 59 accession numbers, but 
the 5 remaining dates did not match exactly; and 

• nearly all of the reports—57 of 59—that we randomly selected from NTIS’ 
warehouse shelves were accurately recorded either in STAR or in the 
index card drawers. 

However, 40 of the 75 reports randomly selected from the drawers were 
also found in STAR. This duplication was not expected because NTIS 
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Appendix I Scope and 
Methodology 


officials said they make their entries into the database directly from the 
document and no longer create an index card. We did not anticipate 
finding cards in the drawers that were evidently created after 1964, which 
is the year NTIS began the database that was later incorporated into STAR. 
However, an NTIS official in the Information Services Division said that 
during the period of 1964-80, NTIS created cards as well as an electronic 
record. They did this because at that time not all employees who needed 
to work with the database had the easy electronic access to it that is now 
routinely provided by personal computers. This double-counting of reports 
in the two systems means that our earlier estimate of 1.5 million reports 
recorded in the index card drawers was overstated by about 53 percent. 
Therefore, we reduced our estimate of the number of reports recorded on 
index cards accordingly, from 1.5 million to 700,000. 

Of the remaining 35 unduplicated reports, NTIS was not able to locate 14 
of them. Thus, we reduced by 40 percent the estimated number of reports 
that do not duplicate the STAR file and that NTIS can find and provide to 
customers—from 700,000 to 420,000 reports. This does not mean, 
however, that NTIS lost nearly 300,000 reports from its repository, since 10 
of the 14 cards in the sample (representing an estimated 200,000 cards in 
the entire index file) were “source filled”—that is, NTIS retains only a 
bibliographic record (not the report) and asks the issuing organization to 
send the report directly to the customer, if it is available. The Information 
Services Division official said that these reports may no longer be 
available from the sources that originally issued them. 

Adding the 420,000 reports from the index cards to the 2.04 million reports 
recorded in STAR, we estimate that NTIS has about 2.46 million reports 
that it can find in the repository or at other agencies. This total is 
understated to the extent that subscriptions are entered annually in STAR 
even though a subscription actually represents the dissemination of more 
than one document over the course of a year. 

To determine the age of NTIS’ research holdings, we used the reports’ date 
of publication in the STAR database as of mid-November 2000 to sort the 
reports into four age strata: 2000; 1997 to 1999; 1989 to 1996; and 1988 or 
earlier. For the 1988 or earlier category, we also added the estimated 
number of reports recorded on index cards. 

To determine the demand for NTIS’ research holdings, we analyzed 
various NTIS statistical and financial data on sales of reports, including the 
financial statement audit reports for fiscal year 1997 through the draft 


Page 29 


GAO-01-490 NTIS Technical Reports 




Appendix I Scope and 
Methodology 


report for fiscal year 2000. Sales data came primarily from NTIS’ database 
called “CIS.PUB,” which has records of all sales of reports since May 1995. 

To determine if the reports in NTIS’ repository were readily available 
elsewhere, we first randomly selected 600 report accession numbers from 
the repository—525 recorded in STAR and an additional 75 accession 
numbers from the index card drawers. These 600 reports consisted of 150 
randomly selected from each of our 4 age strata. The 150 in the oldest 
category (1988 or earlier) consisted of 75 randomly selected from the 
approximately 1.4 million reports in that age category in STAR plus 75 
randomly selected from those recorded in index card drawers. 

Because we selected a sample of reports, all of the results from this 
sample are estimates that are subject to sampling errors. These sampling 
errors measure the extent to which samples of this size and structure are 
likely to differ from the population they represent. Each of the sample 
estimates is surrounded by a 95-percent confidence interval, indicating 
that we can be 95 percent confident that the interval contains the actual 
population value. Unless otherwise noted, 1 the 95-percent confidence 
intervals for percentage estimates throughout this report do not exceed 
plus or minus 10 percentage points around the estimate. 

Next, we searched the Internet for each of the 600 reports in our sample, 
using the following steps (in this order): (1) on the issuing organization’s 
Web site and, if unavailable there, through e-mail to its Office of Public 
Information or equivalent; (2) on the Government Printing Office’s Web 
site; (3) on the FirstGov.gov Web site; and (4) on Google.corn’s Web site. 2 
Because organization officials may be more responsive to an e-mail 
request from us than they would be from the public, we did not use our 
computers to send the e-mail requesting reports. Thus, the agencies’ 
responses to our e-mails should be similar to what any citizen would 
expect who inquires about how to obtain a report. If an official at any of 
these sources told us the report was available for a fee, we assumed it was 
available and did not order the report because of the time and expense 
involved. If we found in a previous step (e.g., searching an organization’s 


1 For our estimate that 44 percent of reports that sold in all age strata since 1995 were 
available elsewhere, the 95-percent confidence interval is plus or minus 15 percentage 
points around the estimate. 

2 Google.com was selected (rather than another search engine) after discussions with the 
requesters’ offices. 
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Appendix I Scope and 
Methodology 


Web site) that the report was available, we still attempted to find the 
report at all remaining sources (except the Office of Public Information). 

Finally, we used our sample results to estimate what portion of the total 
population of NTIS reports was available through the four different 
sources discussed above in each age category. We performed a separate 
analysis of our sample results to show whether reports that had sold one 
or more copies since 1995 were more or less readily available through the 
same four sources, compared with reports that had not sold. 

To determine whether federal agencies were sending their research 
reports to NTIS for sale to the public during calendar year 2000 as required 
by law, we asked NTIS officials to provide automated data from the STAR 
database that would allow us to identify all of the reports in their 
repository that they had harvested from the issuing agency’s Web site 
during that period. Then, to determine why agencies had not provided the 
reports, we asked officials at the seven agencies with the most harvested 
reports why they had not sent their reports to NTIS. These seven agencies 
represented more than half of all harvested reports. Although we did not 
review the content of all 1,127 harvested reports to determine if they 
should have been sent to NTIS, we reviewed the report titles to determine 
if they appeared to be scientific, technical, engineering, or business-related 
information. We did not evaluate whether each of the harvested reports 
was covered under the law. 

We conducted our review from September 2000 to April 2001, in 
accordance with generally accepted government auditing standards. 
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Appendix II: Comments From the 
Department of Commerce 


THE SECRETARY OF COMMERCE 

Washington, D.C. 20230 

APR 2 6 2001 


Mr. Joel Willemssen 
Managing Director 
Information Technology Issues 
General Accounting Office 
Washington, D.C. 20548 

Dear Mr. Willemssen: 

Thank you for the opportunity to review and comment on your draft report on information 
dissemination by the National Technical Information Service (NTIS). On the whole, I believe 
your report properly identifies the key policy issues — how to define scientific and technical 
information, whether there is still a need for a central repository, how long reports should be 
maintained, and what is the proper business model for NTIS. 

Of these, the key question is whether there is a need for a central repository “in view of the 
increasing availability of newer publications from sources other than NTIS” (page 21). Again, I 
agree with your assessment that this depends on whether the information will be permanently 
maintained by agencies and whether it will be easy to locate absent the kind of bibliographic 
control NTIS provides. 

As to permanence, the federal agencies are only just beginning to come to grips with the 
complex issues surrounding digital preservation under the leadership of the Library of Congress’ 
National Digital Information Infrastructure and Preservation Program. As to ease of locating 
information, your report demonstrates the uncertain results currently yielded by conventional 
search engines even when a document is known to be on an agency web site and is known by 
title, author, date, and issuing organization (pages 14 and 28). Without proper bibliographic 
control, the problems facing a researcher who lacks such basic information and wants to know 
what the government as a whole has produced on a given subject and where to find it would be 
daunting. You are quite right to highlight these important issues. 

I generally concur with the thrust of your recommendations and, pending final review and 
clearance of the report, will determine how best to implement them. I will ask the Acting Under 
Secretary for Technology to determine if there are any practical ways, consistent with NTIS’ 
statutory requirement to be substantially self-sustaining, to make the information in its collection 
more widely available to the general public. 




(512025) 
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